Visual display systems, such as flight simulation systems, are commonly employed to train military and commercial pilots. Conventional visual display systems include one or more screens onto which a video image is projected by one or more projectors containing image sources, such as cathode ray tubes (CRTs). The operator of the visual display system is also generally provided with a control panel and, in some instances, a joystick for providing input to the visual display system in response to the displayed video image. In this regard, the control panel and joystick are often designed to duplicate the controls and displays found in an aircraft. Thus, the operator can simulate the flight of an aircraft, for example, and can respond to the environment as depicted by the visual display.
In order to provide a relatively large display for the operator, the video image produced by the projector of a conventional flight simulation system is generally expanded and displayed upon a screen having a much larger surface area than the surface of the image source. For example, each optical line pair of a conventional projector generally defines a projection angle of 7 to 12 arcminutes. Thus, while a relatively large image is displayed for the operator, the resolution of the image is somewhat diminished by the expansion of the video image.
To improve the effective resolution of the projected video image, systems and methods have been developed to inset a high resolution video image into the displayed video image or background image. The high resolution inset image is generally relatively small and is, for example, positioned at an area of interest, such as the portion of the screen that the operator is currently viewing. The high resolution inset image generally has a small relative size, taking advantage of the human visual system's limited ability to see clear images outside a small area surrounding the line of sight. The small region surrounding the line of sight corresponds to the foveal region of the human eye and, therefore, the corresponding ability to see high detail in the region surrounding the line of sight is typically referred to as foveal vision. This high resolution inset image, which is positioned to generally follow the operator's line of sight, is typically referred to as the Area of Interest (AOI) display. The line of sight including the foveal vision of the operator is generally defined by (1) the direction the operator's head faces, and (2) the direction the operator's eyes face relative to the position of the operator's head.
An area of interest 8 is typically determined by a head/eye tracking device, as shown in FIG. 1. As shown, to track the operator's head position, the head/eye tracking device 10 includes a head tracker sensor 12 mounted above the operator's head by means of a headband 14, helmet, or other securing device worn on the operator's head so that the position of the head/eye tracking device is held fixed in position and orientation relative to the operator's head. To track the position of the operator's eyes, the head/eye tracking device includes a video camera 16 mounted above and forward of the operator's head by means of the headband or other head-mounted securing device. The image capture device tracks the position of the operator's eyes by imaging the pupils of the operator's eyes through a partially reflective/partially transmissive flat mirror 18, which is mounted in front of the operator's eyes. By imaging the operator's pupils through the mirror, the image capture device can be mounted in a position out of the operator's visual field of view. In operation, before the head/eye tracking device 10 can track position of the operator's head and eyes, the head/eye tracking device must be calibrated with the screen upon which the video image produced by the projector is displayed. According to one conventional method of calibrating the head/eye tracking device, a plurality of calibration points are displayed upon the screen, such as nine calibration points displayed in a 3×3 matrix. Then, to calibrate the head tracker sensor 12 of the head/eye tracking device, the operator moves his head and, thus, the head tracker sensor such that one of the calibration points is in line with a fixed reference point attached to the head/eye tracker. A data point is then recorded based upon the position of the head tracker sensor. The operator then repeats the process for each of the calibration points displayed upon the screen, with a data point recorded at each calibration point. Once a data point has been recorded for all calibration points, a head tracking calibration function is calculated based upon the recorded data points.
Either before or after calibrating the head tracker sensor, the image capture device of the head/eye tracker must be calibrated with the screen so that the head/eye tracking device can accurately track the position of the operator's eyes relative to the head tracker sensor. To calibrate the image capture device, the calibration points are again displayed upon the screen. The operator then orients his head and, thus the head tracker sensor, in a fixed position, preferably such that the operator's line of sight intersects a center point of the screen. Holding his head in the fixed position, the operator moves his eyes such that the operator's line of sight intersects one of the calibration points. A data point is then recorded for the calibration point based upon an image of the operator's pupils taken by the image capture device. In this regard, the data point can include information such as pupil position in both the X and Y directions, as well as corneal reflection in both the X and Y directions. Also as before, the operator repeats the process of moving his eyes to each calibration point and recording a data point for each calibration point, all while holding his head in the fixed position. And after the data points have been recorded for all of the calibration points, an eye tracking calibration function is calculated based upon the recorded data points, e.g., (eye pitch, eye yaw)=f(input data from tracker).
After calculating the calibration functions, the head/eye tracking device tracks the position of the operator's head based upon the position of the operator's head as determined by the head tracker sensor, and based upon the head tracking calibration function. Similarly, the head/eye tracking device tracks the position of the operator's eyes as determined by the image capture device, and based upon the eye tracking calibration function. And based upon the position of the operator's head and pupils at any given time, the area of interest is selected relative to the screen and thereafter displayed, such as in line with the operator's line of sight. The high resolution inset image can then be projected within the area of interest to thereby be within the operator's foveal vision.
Whereas conventional head/eye tracking devices are adequate in tracking the head and eye position of the operator, such conventional tracking devices have drawbacks. The partially reflective/partially transmissive flat mirror 18, which is mounted in front of the operator's eyes, is visually distracting for the operator. In this regard, the mirror is an unnatural obstruction in the operator's visual field of view, limiting the operator's feeling of visual immersion in a virtual reality application, such as that found in flight simulation systems.
In addition to the structural drawback of the partially reflective/partially transmissive flat mirror 18, the method by which conventional head/eye tracking devices are calibrated also has drawbacks. As stated before, either the operator's eyes or head must remain in a fixed position for a period of time during various stages of calibrating the head/eye tracking device. In this regard, the calibration method is prone to errors caused by inadvertent movement of either the operator's eyes or head during a time in which the eyes or head must remain in a fixed position for complete accuracy. Due to the time period the operator is required to keep either his eyes or head in the fixed position, the operator can have a tendency to fatigue, which can cause the operator's eyes or head to inadvertently move.